A computer network typically includes a collection of interconnected computing devices that exchange data and share resources. The devices may include, for example, web servers, database servers, file servers, routers, printers, end-user computers and other devices. The variety of devices may execute a myriad of different services and communication protocols. Each of the different services and communication protocols exposes the network to different security vulnerabilities.
Conventional techniques for detecting network attacks use pattern matching. In particular, an intrusion detection and prevention device (IDP) applies regular expressions or sub-string matches to detect defined patterns within a data stream. Multiple patterns may be used in an attempt to improve the accuracy of the attack detection. In order to improve the probability of detecting an attack, the IDP may attempt to identify the type of software application and protocol associated with the data stream. Based on the identification, the IDP selects the appropriate patterns to apply in order to detect a network attack, such as viruses or other malicious activity.
Traffic encapsulation is becoming more popular. Encapsulation may occur at any layer of the open systems interconnection (OSI) networking model. The purpose could be encryption (e.g. secure sockets layer (SSL), network tunneling (e.g. generic routing encapsulation (GRE)), or simply protocol reusing (e.g. KAZAA over hypertext transfer protocol (HTTP)). In some cases, multiple levels of encapsulation are used to convey network traffic. For example, a network tunnel may carry traffic that itself encapsulates multiple sub-tunnels. In this case, a single packet for the network tunnel may include sub-packets corresponding to the sub-tunnels, and each of the sub-packets contains a header and a payload. This disclosure refers to the packet comprising the sub-packets as an “outer packet” of an “outer session” and the sub-packets as “inner packets” of “inner sessions” or “encapsulated sessions.” Accurately detecting network attacks within encapsulated network traffic represents a challenge to intrusion detection systems.